During the past decade, there was an exponential growth in the number of websites available. Automatic website categorization systems can help to manage these immense amounts of content, making search tasks and recommendation easier. However, most websites have a significant proportion of visual content that conventional, text-based web mining systems can not handle. In this paper, we present a novel hybrid CBR framework designed to perform image-based website categorization. Our system incorporates state-of-the-art deep learning techniques which help attain high accuracy rates. In addition, the system was designed with the goal of minimizing computational costs.